Unsupervised Learning (PCA)

Unsupervised Learning

• In supervised learning we use a set of features, \(\mathbf{X} = \{X_1,X_2,\ldots,X_p\}\), to model an outcome variable \(Y\)
• In unsupervised learning we have \(\mathbf{X}\), but no natural outcome to model
  • Our goals shift towards finding patterns, such as hidden subgroups or clusters in the data

Principle Components Analysis (PCA)

• The goal of PCA is construct a lower-dimensional representation of the \(p\)-dimensional matrix \(\mathbf{X}\) that retains the data’s most important characteristics
• Consider a situation where we have \(p = 10\) variables, and \(n = 50\) observations (for example US state data)
  • Most graphs have an upper limit of 3-4 variables that can be displayed
  • With \(p = 10\) there are too many graphs to exhaustively look at each unique combination of variables
• PCA operates under the assumption that not all variables are equally interesting
  • Some variables might be correlated and actually represent the same underlying factor
  • Some variables might not contain much variation (there is little distinction between observations)

Dimensions

• To understand PCA we need to understand dimensions, consider variables \(X_1\) and \(X_2\), whose relationship can be displayed in two dimensions:

Dimensions

• Most of the variation is from left-to-right, if we ignored variation in \(X_2\) we still have a pretty good representation of the data using only one dimension:

Dimensions

• The last example was technically incorrect because the variables were on different scales, but the concept is important and can be applied to dimensions that are correlated:

Dimensions

• In panel A, projecting each point onto the red line provides a good one-dimensional representation of the data
  • This newly derived dimension is known as a principle component, it is a linear combination of the original variables
  • This principle component explains over 90% of the variability in the data!
• In panel B, projecting each point onto the red line doesn’t provide a good one-dimensional representation of the data

Dimensions

The same idea extends to three-dimensions, we can imagine projecting these three-dimensional points onto a line and retaining the essence of the data:

Dimensions

The same idea extends to three-dimensions, we can imagine projecting these three-dimensional points onto a plane and retaining the essence of the data:

Back to PCA

How do we construct these new dimensions? Answer: Linear algebra

• Let \(\mathbf{X}\) be an \(n\) by \(p\) matrix whose columns have been standardized (think \(z\)-scores)
• Then \(\mathbf{C} = \tfrac{1}{(n-1)} \mathbf{X}^T\mathbf{X}\) is the \(p\) by \(p\) covariance matrix
• Because \(\mathbf{C}\) is symmetric, it can be decomposed: \(\mathbf{C} = \mathbf{V}\mathbf{L}\mathbf{V}^T\)
  • \(\mathbf{V}\) is a matrix of eigenvectors, \(\mathbf{L}\) is a diagonal matrix of eigenvalues
  • The \(j^{th}\) column of \(\mathbf{X}\mathbf{V}\) is the \(j^{th}\) principle component
  • The elements of \(\mathbf{L}\) are the eigenvalues they are tied to the importance (variability explained) by each principle component

What is Important?

• For the purposes of this class, you don’t need to be familiar with any linear algebra, the concept of principle components being lower dimensional representations of a higher dimensional matrix
• I also expect you to know how to apply PCA, so we’ll go through a more formal example using the famous Iris data:

Iris Example

Inspecting the variables in the Iris data, it is clear that dimension reduction might be possible:

Iris Example

prcomp(iris[,-5], scale = TRUE)

## Standard deviations (1, .., p=4):
## [1] 1.7083611 0.9560494 0.3830886 0.1439265
## 
## Rotation (n x k) = (4 x 4):
##                     PC1         PC2        PC3        PC4
## Sepal.Length  0.5210659 -0.37741762  0.7195664  0.2612863
## Sepal.Width  -0.2693474 -0.92329566 -0.2443818 -0.1235096
## Petal.Length  0.5804131 -0.02449161 -0.1421264 -0.8014492
## Petal.Width   0.5648565 -0.06694199 -0.6342727  0.5235971

• The “Standard deviations” are the square roots of the eigenvalues. They describe how much of the variation from the original data is described by that principle component.
• The “Rotations” are the principle components themselves. They are the linear combination of the original variables that represents the new dimension

Iris Example

After performing the eigenvalue decomposition, we see that ~96% of the variation in the data can be retained by the first two principle components. This is calculated from the eigenvalues, for example: \(1.71^2/(1.71^2 + .956^2 + .383^2 + .144^2) = 0.73\)

Iris Example

Since the first two principle components contain ~96% of the variation, we can effectively summarize the data using just those dimensions:

Iris Example

We can also interpret the principle components as new, derived variables:

• The first component is a contrast between overall petal size + sepal length and sepal width
• The second component is overall sepal size

Clustering

• In the Iris example, we ignored the variable “Species” when applying PCA, but the previous plot shows that first two principle components could be used to group the observations
• If we somehow knew that there were three types of iris in the data, we could do a pretty good job identifying them just by inspecting the PCA results
• Taking a more systematic approach, we could formally group the observations using clustering